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Abstract 

In  this  paper  we  present  new  uncalibrated  control  schemes  for  vision- 
guided  robotic  tracking  of  a  moving  target  using  a  moving  camera. 
These  control  methods  are  applied  to  an  uncalibrated  robotic  system 
with  eye-in-hand  visual  feedback.  Without  a  priori  knowledge  of  the 
robot's  kinematic  model  or  camera  calibration,  the  system  is  able 
to  track  a  moving  object  through  a  variety  of  motions  and  main¬ 
tain  the  object's  image  features  in  a  desired  position  in  the  image 
plane.  These  control  schemes  estimate  the  system  Jacobian  as  well 
as  changes  in  target  features  due  to  target  motion.  Four  novel  strate¬ 
gies  are  simulated  and  a  variety  of  parameters  are  investigated  with 
respect  to  performance.  Simulation  results  suggest  that  a  Gauss- 
Newton  method  utilizing  a  partitioned  Broy  den's  method  for  model 
estimation  provides  the  best  steady-state  tracking  behavior. 

KEY  WORDS — uncalibrated  eye-in-hand  visual  servoing, 
nonlinear  least  squares,  Jacobian  estimation 


1.  Introduction 


In  this  paper  we  develop  a  model-independent,  vision-guided, 
robotic  control  method.  The  controller  presented  is  a  recur¬ 
sive  Gauss-Newton  method  and  uses  nonlinear  least-squares 
optimization.  The  combined  camera/robot  model  is  approx¬ 
imated  in  a  dynamic  Jacobian  estimation  strategy,  allowing 
servo  control  to  be  applied  to  systems  without  requiring  a 
robot  kinematic  model  or  a  calibrated  camera  model.  Error 
velocity  estimation  is  done  simultaneously  with  Jacobian  es¬ 
timation  in  a  partitioned  matrix  method.  The  control  method 
is  completely  independent  of  the  type  of  robot,  the  type  of 
camera,  and  number  of  cameras. 

Much  work  has  been  done  in  developing  visual  servoing 
systems  for  robot  control  resulting  in  a  variety  of  approaches 
to  the  servoing  problem.  The  majority  of  the  resulting  meth¬ 
ods,  however,  require  a  priori  knowledge  of  the  system  includ¬ 
ing  the  kinematic  structure  and  the  camera  parameters.  Using 
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a  model-dependent  system  also  demands  calibration  of  the 
robot  and  vision  system,  or  recalibration  due  to  the  sensitivity 
of  the  system  to  disturbances.  These  activities  can  be  both 
difficult  and  time-consuming,  or  perhaps  unfeasible  in  an  un¬ 
structured  or  changing  environment.  To  our  knowledge,  this 
is  the  first  method  that  addresses  the  uncalibrated  eye-in-hand 
visual  servoing  problem  for  a  moving  target. 

2.  Background 

Benefits  of  using  eye-in-hand  camera  arrangements  include 
improved  target  recognition  and  inspection  resulting  from  lo¬ 
calization  described  by  Chaumette,  Rives,  and  Espiau  (1991). 
Jang  and  Bien  (1991)  contend  that  effective  resolution  is  in¬ 
creased,  the  problem  of  occlusion  is  solved,  and  an  image 
nearly  free  of  parallax  error  can  be  obtained  using  an  eye-in¬ 
hand  camera.  Static  cameras  can  be  limited  in  their  abilities 
due  to  limited  depth  of  field  and  spatial  resolution,  problems 
which  can  be  relieved  by  using  eye-in-hand  cameras  as  indi¬ 
cated  by  Papanikolopoulos,  Khosla,  and  Kanade  (1993). 

2.1 .  Use  of  Eye-In-Hand  Camera 

The  majority  of  eye-in-hand  visual  servoing  controllers  fall 
under  two  categories  which  could  limit  their  use.  Either  they 
are  not  model-independent  with  regards  to  either  the  camera 
or  the  robot,  or  they  require  that  the  target  is  static. 

Hashimoto  and  Noritsugu  (1999)  developed  a  linearized 
observer  to  estimate  target  velocity  in  their  model-dependent 
controller.  Allotta  and  Colombo  (1999)  also  use  linear  ap¬ 
proximations  in  an  affine  camera-object  interaction  model 
with  the  robot  kinematic  model  assumed  known.  Baeten  and 
De  Schutter  (1999)  use  an  eye-in-hand  feedforward  controller 
in  conjunction  with  force  control  for  planar  contour  follow¬ 
ing.  Both  a  camera  model  and  a  robot  model  are  required  in 
this  case. 

Algorithms  using  a  PI  controller  or  a  pole  assignment  con¬ 
troller,  both  in  combination  with  a  steady- state  Kalman  filter, 
are  implemented  by  Papanikolopoulos,  Khosla,  and  Kanade 

805 


Downloaded  from  http://ijr.sagepub.com  at  US  NAVAL  ACADEMY  LIBRARY  on  December  9,  2008 


806  THE  INTERNATIONAL  JOURNAL  OF  ROBOTICS  RESEARCH  /  October-November  2003 


(1993).  In  each  method,  control  output  is  fed  to  a  Cartesian 
positioning  system  which  requires  a  kinematic  model. 

Yoshimi  and  Allen  (1994)  implement  the  geometric  effect 
of  rotational  invariance  to  approximate  the  image  Jacobian.  In 
addition  to  requiring  knowledge  of  the  robot  Jacobian,  their 
algorithm  inherently  works  for  only  a  static  target. 

Cretual,  Chaumette,  and  Bouthemy  (1991)  develop  a  con¬ 
trol  algorithm  for  tracking  a  moving  target  with  no  model  of 
the  image  transformation.  The  robot’s  vision  system,  however, 
consists  of  a  camera  with  only  two  degrees  of  freedom  (DoF), 
pan  and  tilt,  which  are  controlled  by  an  image  error  minimiza¬ 
tion.  Depth  is  not  considered  in  the  process,  as  tracking  is  only 
done  in  the  image  plane,  not  in  Cartesian  space. 

Flandin,  Chaumette,  and  Marchand  (2000)  build  upon  the 
work  done  by  Cretual,  Chaumette,  and  Bouthemy  (1991)  to 
create  a  visual  servo  controller  for  a  6-DoF  robot.  In  this  case, 
the  global  positioning  is  done  by  a  static,  global  camera,  and 
the  rotational  positioning  is  servoed  by  another,  independent 
controller.  These  algorithms  do  require  knowledge  of  the  kine¬ 
matic  robot  Jacobian. 

Oh  and  Allen  (2001)  present  real-time  experiments  track¬ 
ing  people  and  robot  hands  (moving  targets)  using  a  3 -DoF 
gantry  robot  with  a  2-DoF  pan-tilt  unit.  The  approach  par¬ 
titions  the  axes  based  on  dynamic  performance  to  servo  the 
camera.  The  controller  uses  the  kinematic  and  dynamic  at¬ 
tributes  of  each  DoF  to  increase  tracking  performance. 

Corke  and  Hutchinson  (2001)  develop  a  decoupled  image- 
based  controller  to  handle  pathological  image-based  visual 
servoing  problems  for  the  eye-in-hand  case.  They  also  employ 
a  potential  field  technique  to  prevent  features  from  leaving  the 
image  plane.  The  tasks  studied  involve  static  targets. 

Malis  and  Chaumette  (2002)  and  Malis  (2002)  discuss  a 
task-function  approach  to  image-based  model-free  visual  ser¬ 
voing.  This  work  uses  information  from  a  eye-in-hand  camera 
to  servo  the  robot  into  a  desired  orientation  with  respect  to  a 
static  target.  In  Malis  (2002)  the  controller  is  able  to  achieve 
straight-line  Cartesian  motion  of  the  robot-mounted  camera. 
It  is  important  to  note  that  the  reference  to  “model-free”  vi¬ 
sual  servoing  in  these  works  refers  to  the  fact  that  there  is 
no  known  model  of  the  target  object  and  the  camera  models 
are  uncalibrated.  The  method  is  robust  with  respect  to  camera 
calibration  errors,  but  a  well-calibrated  robot  is  used. 

Several  methods  have  been  developed  using  a  rank  one 
update  estimation  of  a  composite  Jacobian,  known  as  a  Broy- 
den  estimator.  The  Jacobian  estimate  is  employed  as  part  of 
a  Gauss-Newton  method  to  minimize  squared  error  in  the 
image  plane.  The  Broyden  estimator  develops  an  on-line  esti¬ 
mate  of  the  system  model,  relating  changes  in  image  features 
to  changes  in  joint  variables.  This  means  that  no  kinematic 
or  camera  model  is  used.  Knowledge  or  estimation  of  indi¬ 
vidual  kinematic  and  camera  parameters  are  not  necessary. 
Piepmeier  (1999)  builds  on  work  carried  out  by  Jagersand, 
Fuentes,  and  Nelson  (1997)  and  Hosada  and  Asada  (1994) 
to  develop  a  dynamic  Gauss-Newton  method  of  visual  servo 


control  for  tracking  a  moving  target.  A  recursive  least-squares 
(RLS)  algorithm  is  implemented  for  Jacobian  estimation  to 
provide  a  robust  control  method  even  in  the  presence  of  sys¬ 
tem  and  measurement  noise.  The  controller  does  not,  however, 
permit  the  application  of  a  moving  camera.  This  paper  extends 
the  controller  developed  for  the  fixed  camera  case  to  solve  the 
moving  camera  problem. 

2.2.  Uncalibrated  Control  for  Fixed  Camera  Visual 
Servoing 

In  this  section  we  present  the  fixed  camera  case  for  reference. 
In  Section  3  it  is  modified  for  the  moving  camera  case.  First 
the  dynamic  Gauss-Newton  controller  is  presented  followed 
by  the  dynamic  Jacobian  estimator. 

2.2.7.  Dynamic  Gauss-Newton  Method 

The  dynamic  Gauss-Newton  method  minimizes  a  time- 
varying  objective  function  based  on  errors  in  the  image  plane. 
If  the  desired  behavior  is  simple  target  following,  the  error 
function  in  the  image  plane  for  a  moving  target  at  position 
y*  (t)  and  an  end-effector  at  position  y  (6)  is  the  residual 
error 

f(0,t)  =  y(0)-y*(t)  (1) 

where  9  represents  the  joint  angles  and  t  represents  time.  The 
objective  function  to  be  minimized  is  the  squared  error: 

F(0,t)  =  ^fT(0,t)f(0,t).  (2) 

Objective  function  (2)  can  be  minimized  using  the  dynamic 
Gauss-Newton  algorithm  (Piepmeier,  McMurray,  and  Lipkin 
1999a).  Let  Jk  represent  the  kth  approximation  to  the  Jacobian 
H  =  Jk.  The  dynamic  Gauss-Newton  method  computes  the 
joint  angles  iteratively  to  minimize  the  objective  function  (2) 

0w  =  ek-(%jtyl  %(fk  +  ^ht)  (3) 

where  ht  =  tk  —  tk_x  is  the  time  increment.  The  term  d-^ht 

predicts  the  change  in  the  error  function  for  the  next  itera¬ 
tion.  This  term  is  a  critical  addition  to  the  work  presented 
by  Jagersand,  Fuentes,  and  Nelson  (1997)  and  Hosada  and 
Asada  (1994)  that  enables  the  robot  to  servo  to  and  track  tar¬ 
get  motion.  Note  that  eq.  (3)  is  the  controller  used  to  compute 
desired  changes  in  joint  angles.  It  is  assumed  that  individual 
joint  level  controllers  are  used  to  effect  the  changes. 

A  dynamic  RLS  estimation  is  used  to  provide  on-line  esti¬ 
mates  of  the  Jacobian  Jk  as  discussed  in  Piepmeier,  McMur¬ 
ray,  and  Lipkin  (1999b)  and  as  follows. 
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2.2.2.  Dynamic  Jacobian  Estimation:  Recursive 
Least-Squares 

Since  the  robot  model  is  assumed  to  be  unknown,  a  RLS  al¬ 
gorithm  is  used  to  estimate  the  system  Jacobian.  This  is  ac¬ 
complished  by  minimizing  a  weighted  sum  of  the  changes  in 
the  affine  model  of  the  error  in  the  image  plane.  The  affine 
model1  of  the  error  function  /  (0,  t)  is  denoted  as  m(0,  t)  and 
expansion  about  the  kth  data  point  gives 

mk(0,  t)  =  f  (6t,  4)  +  Jt(6  -9k)  +  ^(t-  4) .  (4) 

Ot 

As  shown  in  Haykin  (1991)  an  exponentially  weighted  RLS 
algorithm  that  minimizes  a  weighted  sum  of  the  changes  in 
the  affine  model  over  time  can  be  used  to  recursively  estimate 
the  system  Jacobian. 


is  zero  at  the  ( k  —  l)th  data  point.  Note  that  by  minimizing  a 
weighted  sum  of  the  changes  in  the  affine  error  models,  the 
algorithm  will  generally  not  satisfy  the  secant  condition  as  in 
Broyden’s  method  (Dennis  and  Schnabel  1983).  Hosoda  and 
Asada  (1994)  and  Hosoda,  Igarashi,  and  Asada  (1998)  imple¬ 
ment  a  RLS  Jacobian  estimation  scheme  similar  to  Broyden’s 
method  with  addition  of  a  forgetting  factor.  In  Piepmeier,  Mc- 
Murray,  and  Lipkin  ( 1 999b)  it  has  been  shown  that  this  method 
is  a  RLS  technique,  a  type  of  Kalman  filter.  The  RLS  algorithm 
displays  greater  stability  than  Broyden’s  method  for  Jacobian 
estimation  by  considering  data  over  a  period  of  time  instead 
of  just  the  previous  iteration. 

The  Jacobian  estimation  in  eq.  (7)  requires  the  (partial) 
velocity  error  term  ^ ,  which  can  be  directly  estimated  from 
the  target  image  feature  vector  when  a  static  camera  is  used 
with,  for  example,  first-order  differencing: 


min  8k 
A /no¬ 


where 


k-i 

1 '  II  Am*,  ||2 

i= 0 


mk  (i Oi ,  U)  mL  ( 0i ,  U) 

dfk 

fk~  fi~  dt  ~~ 


Jkhk 


(5) 

(6) 


M  =  (d(y(Q)-y*(t))\  „  (y;~ylA 
dt  \  dt  A  \  ht  )' 

However,  in  the  moving  camera  case  the  image  target  features 
are  a  function  of  0  and  t,  y*  =  y*  (0,  t).  This  makes  a  similar 
direct  calculation  of  the  time  partial  derivative  of  error  unfea¬ 
sible.  This  is  resolved  using  the  partitioned  recursive  Jacobian 
estimation  scheme  in  the  following  section. 


hki  =  (0k  -  Oi) 

and  where  the  weighting  factor  is  0  <  A,  <  1  and  the  unknown 
variables  are  the  elements  of  Jk.  The  problem  is  solved  in 
Appendix  A  to  yield 

h  =  1-1  +  (a/  -  J^he  - 

(X  +  hlP^hoY'hlP^  (7) 

Pk  =  ^(pk-,-Pk-lhe(k  +  h'lPk_lheylhlPk_iy?,) 


3.  Uncalibrated  Control  for  Eye-in-Hand  Visual 
Servoing 

With  an  eye-in-hand  system,  changes  in  image  features  of 
a  target  object  may  be  due  either  to  target  motion  or  robot 
motion.  In  this  section  we  develop  a  nonlinear  least- squares 
optimization  method  for  a  time- varying,  coupled  system  and 
we  introduce: 

1.  a  partitioned  Broyden’s  update  for  the  (partial)  error 
velocity  estimation; 


where  he  =  0k  —  0k_u  A /  =  fk  —  fk_x.  Equations  (7)  and 
(8)  define  the  recursive  update  for  Jk.  At  each  iteration,  the 
calculation  of  Jk  using  eqs.  (7)  and  (8)  solves  the  minimiza¬ 
tion  of  eq.  (5).  The  Jacobian  estimate  is  used  in  the  dynamic 
Gauss-Newton  method  of  Section  2.2. 1  to  determine  the  joint 
angles  0k  that  track  the  target.  The  parameter  k  can  be  tuned 
to  average  in  more  or  less  of  the  previous  data  in  the  mini¬ 
mization.  The  approximation  A  is  often  used  to  estimate  the 
effective  number  of  terms  being  minimized.  A  k  close  to  1 
results  in  a  filter  with  a  longer  memory. 

Equation  (7)  bears  some  similarity  to  Broyden’s  method 
of  Jacobian  estimation  as  employed  by  Jagersand,  Luentes, 
and  Nelson  (1997).  Broyden’s  method  (Dennis  and  Schnabel 
1983)  is  a  Jacobian  estimation  scheme  that  satisfies  the  secant 
condition,  i.e.  the  change  in  the  mk_x  and  mk  affine  models 

1 .  An  affine  model  is  a  linear  model  that  does  not  necessarily  pass  through 

the  origin. 


2.  recursive  and  non-recursive  forms  of  a  Gauss-Newton 
method  implementing  the  partitioned  Broyden  update; 

3.  a  correction  scheme  for  the  error  velocity  estimation 
using  the  partitioned  Broyden  result  as  a  predictor. 

The  partitioned  Broyden  update  implements  a  simultane¬ 
ous  dynamic  RLS  estimation  of  the  Jacobian  and  the  error 
velocity.  The  error  velocity  estimation  corrector  is  based  on 
the  total  time  derivative  of  the  error  function.  Various  con¬ 
trollers  are  created  in  the  following  section  by  combining  the 
two  Gauss-Newton  forms  with  the  error  velocity  correction. 

3.1.  A  Partitioned  Broydenys  Method 
The  dynamic  Broyden’s  method  presented  in  Piepmeier,  Mc- 
Murray,  and  Lipkin  (1999a)  requires  that  the  target’s  image 
position  y  *  be  independent  of  robot  joint  position  0 ,  and  only  a 
function  of  time.  In  the  moving  camera  case,  this  is  no  longer 
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valid,  because  y*  =  In  successive  iterations,  the 

target  image  position  may  change  due  to  camera  movement, 
even  if  the  actual  target  is  stationary.  This  makes  estimation 
of  the  error  velocity  necessary. 

The  estimation  of  the  error  velocity  can  be  incorporated 
into  a  Broyden  Jacobian  estimation  method  using  a  partitioned 
matrix  to  rewrite  eq.  (6)  as 

AW2fc(i_  1)  Yfk  fi— ll  *4^4(7— 1)  (9) 


where 


The  term  is  the  estimated/predicted  value  of  the  error 

velocity  This  partitioned  Broyden’s  method  uses  RLS  to 
minimize  a  weighted  sum  of  the  squared  differences  between 
the  current  and  previous  affine  models  for  each  iteration  of 
(i 0,t )  =  (ft_i ,  ti_])  to  determine  new  Jacobian  and  error  ve¬ 
locity  approximations.  Varying  the  parameter  X  between  0  and 
1  changes  the  memory  of  the  scheme,  with  values  closer  to  1 
resulting  in  longer  memory,  implying  that  a  greater  number  of 
previous  Jacobian  and  error  velocity  values  have  a  significant 
effect  on  the  new  values. 


*4 

hk(i- 1) 


U  t  ] 

'  ( ek  -  $_,) 

(4  -  U- 1) 


so  the  Jacobian  is  augmented  by  the  error  velocity  and  the 
joint  angle  differences  are  augmented  by  a  time  difference. 
Noting  the  similarity  of  eqs.  (6)  and  (9),  the  RLS  solution  for 
the  minimization  of  8k  in  eq.  (5)  is  given  by 


3.2.  Recursive  Gauss-Newton  Method 

In  order  to  provide  a  filtering  action  to  the  joint  change  calcula¬ 
tion,  the  Gauss-Newton  method  is  extended  to  an  RLS  formu¬ 
lation  with  exponential  weighting.  In  this  case  the  weighted 
sum  being  minimized  uses  the  affine  error  models  evaluated 
at  the  next  data  point 


Jk  —  *4—1  T 

a-K* 


(a/  -  +_,/?)  (x  +  ATPt_iA)_1  hTPk_ , 
-i  -  A-i h  (a  +  Ph-th)  ’  hJPkN 


(10) 

(11) 


where 


(ft  —  ft-  i) 
(4  —  4-i) 


min  %  =  1  || m,  (6k+u  4+i)l|2  (12) 

i  =  0 

mi  (ft+i,  4+i )  =  fi  +  *404+1  —  ft)  +  ~~  (4+i  ~  U)  (13) 

ot 

=  fi  +  (4+i  —  4)  +  *4  (ft  —  ft) 

cu 

-  (-*4)  (ft+i  -  ft)  (14) 


While  eqs.  (7)  and  (8)  are  inherently  similar  to  eqs.  (10) 
and  (11),  it  is  important  to  note  the  difference  between  Jk 
used  in  eq.  (7)  and  the  partitioned  Jk  used  in  eq.  (10)  which 
incorporates  both  Jk  and  ^  to  be  used  by  the  controller.  An 
algorithm  for  the  combination  of  the  partitioned  Broyden  up¬ 
date  with  the  (non-recursive)  Gauss-Newton  controller  given 
by  eq.  (3)  in  Section  2.2.1  is  as  follows: 


which  must  be  solved  for  ft+1  given  tM.  This  can  be  rewrit¬ 
ten  as 

min  fyk  =  min  eTWe 
e  =  h  —  Axk 


Algorithm  1:  Gauss-Newton  Controller  with  Partitioned 
Broyden’s  Method  (NGN) 

Given  /  :  Rn  Mw;  ft,  ft  e  R";  J0  e  Mwxn; 


(41 


4  P0  e 


;  A.  e  (0,  1) 


Do  for  k  =  1,2,... 


A /  —  fk  —  fk-u  he  —  ft  —  ft_i,  ht  —  tk  —  4-i 
r  _  ["  (ft  —  ft-i) 

L  &  -  J 

+-i = [  i-i  i  ] 

Jk  =  *4-i  +  /  —  Jk-\h^  ^  +  hT Pk-\h^  hT Pk_ i 

n  =  I  (A_i  -  Pt-ih  (a  +  ftT/u*)  1  hTPk- 1  j 

ft+i = ot  -  (uyl  (/;/, + 4  (/,)t +) 

End  for 


where 


mo  (ft+i»  4+i ) 

y°I 

e  = 

,  W  = 

wfc(ft+i>4+i) 

yk~lI  _ 

/o  +  (4+i  —  4)  +  Jo  (ft  —  ft) 

-  fk  4"  fr  (4+i  ~  4)  +  *4  (ft  —  ft)  _ 

r  -ft  i 


A 


xk  —  (ft+i  —  ft)  — 


The  RLS  solution  for  the  minimization  of  %k  in  eq.  (12)  is 
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given  by 

ft+ 1  =  2ft  —  ft- 1  —  Qk-lJk  (y  I  +  JkQk-\Jk  ^ 

{^fk  +  -  (4+i  ~  4)  +  *4  (ft  —  ft-i)^  (15) 

Qk  =  \{Q^-Q^Jk(Yl  +  JkQ^jf)  JkQk-i^  ■ 


Combining  this  recursive  Gauss-Newton  method  with  the 
partitioned  Broy den’s  method  and  using  the  predicted  value 
of  the  error  velocity  gives  the  following  algorithm: 


Algorithm  2:  Recursive  Gauss-Newton  Method  (RGN) 
Given  f  :Rn  ^  ft,  ft  e  ft  e  Mmxn; 


(4 


lm,  Po  £ 


pn+lxn+l . 


;  Q o  e  Knx";  y,Xe  (0,  1) 


Do  for  k  =  1,2, ... 

A/  =  fk  —  fk-u  he  =  Ok  —  9k- 1,  h,  —  tk  —  4-i 


/! 


-[ 


(ft  _  ft-l) 

(4  —  4-i) 

=  [  jk- 1  ] 

7*  =  /*_,  +  (a/  -  (a  +  hJPk-,h)  1 
A  =  \  (ft_i  -  A-,/1  (a  +  7TA-1^)_1 


.  ^  (P)t  ]  ~  ~Jk 

ft+i  —  2ft  —  ft_i  —  Qk-iJk  (y I  +  JkQk-\Jk ^ 

(a  +  (/r)^  (4+1  —  4)  +  ft  (ft  —  ft-l)^ 

G&  =  ~  ^Gfc-i  —  Qk-iJk  (y^  +  ftG^— iftT)  ftGfc-i^ 
End  for 


Again,  there  is  a  memory  term  in  the  formulation,  ]/ ,  similar  in 
function  to  X,  which  can  be  tuned  to  vary  the  effect  of  previous 
information. 


3.3.  Estimated  Error  Velocity  Correction 

The  partitioned  Broy  den’s  method  uses  RLS  estimation  to 
simultaneously  approximate  the  Jacobian  J  and  the  error  ve¬ 
locity  d-2  because  for  the  moving  camera  case  they  cannot  be 
determined  separately.  The  averaging  process  of  this  method, 
however,  may  not  produce  accurate  error  velocity  values.  The 
estimated  Jacobian  can  also  be  used  in  a  corrector  to  calculate 
new  values  of  the  error  velocity  through  use  of  the  total  time 
derivative: 


d /J  P 

d t  dt  d0  d t  ' 

The  term  d-ft  represents  the  change  in  error  due  to  the  tar¬ 
get  motion  since  the  end-effector  position  is  instantaneously 


fixed.  Introducing  the  approximations  ^  =  A  f/ht,  the  total 
change  in  error,  and  ^  ^  =  ( Jhe )  /  ht ,  the  change  in  the  error 
due  to  end-effector  motion,  then  for  the  4th  increment 

(/)t  =  (A  f-Jkhe)/h,  (16) 

where  is  the  estimated  value  of  (ff)r  Using  the  above 
equation,  an  approximation  of  the  error  velocity  values  can 
be  made  for  comparison  with  those  values  given  by  the  parti¬ 
tioned  Broyden  estimator.  Thus,  a  Gauss-Newton  controller 
(3)  or  the  recursive  Gauss-Newton  controller  (15)  can  either 
implement  the  uncorrected  error  velocity  values  output  by  the 
partitioned  Broyden’s  method  or  the  corrected  error  velocity 
values  given  by  the  total  time  derivative  equation.  In  other 
words,  eq.  (16)  can  be  inserted  in  either  Algorithm  1  or  2  af¬ 
ter  the  Jacobian  estimation.  It  should  be  noted  that  all  four 
control  options  compute  desired  changes  in  joint  angles,  and 
it  is  assumed  that  the  robot’s  individual  joint  level  controllers 
are  able  to  achieve  the  desired  changes. 

4.  Simulation  and  Results 

In  this  section  we  present  a  series  of  simulations  to  evaluate 
the  control  schemes.  First,  the  four  controllers  are  compared 
for  a  simple  translational,  circular  path  for  varying  speeds 
and  A.  Then  with  a  selected  controller  more  difficult  paths  are 
examined. 

4.1.  System  Description 

A  6-DoF  system  is  simulated  using  the  Robotics  and  Machine 
Vision  Toolboxes  developed  by  Corke  (1996)  for  use  in  the 
MATLAB  environment.  The  camera  is  assumed  to  be  coinci¬ 
dent  with  the  final  frame  of  the  Puma  560  manipulator.  The 
dynamics  of  the  robot  are  not  considered  in  this  simulation.  A 
sampling  time  of  ht  =  50  ms  is  used  throughout  for  a  20  Hz 
system  update.  While  vision  latencies  are  not  explicitly  mod¬ 
eled,  it  is  assumed  that  the  vision  data  at  each  update  represent 
the  changes  in  features  for  the  most  recently  commanded  mo¬ 
tion. 

The  target  consists  of  four  planar  feature  points  spaced 
on  a  square  with  5  cm  sides.  The  target’s  initial  location  is 
within  the  camera’s  field  of  view  and  is  approximately  50  cm 
in  front  of  the  camera.  The  image  features  seen  at  this  point 
define  the  target  image.  As  the  target  moves,  the  error  will  be 
minimized  as  the  robot  servos  the  camera  so  that  the  camera 
and  the  target  maintain  constant  relative  positions.  To  start 
the  simulation,  the  robot  is  moved  away  from  the  target.  To 
estimate  the  initial  Jacobian,  each  joint  is  successively  moved 
a  small  amount  and  the  change  in  image  features  is  recorded. 

After  capturing  the  target  image  and  initializing  the  Jaco¬ 
bian,  the  simulation  is  ready  to  begin.  The  target  center  is 
given  a  circular  motion  with  constant  orientation  in  the  fixed 
frame 
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(x,  y,  z)  =  (65,  50  +  10  sin  ( kcoht ) ,  10  cos  ( kcoht ))  (17) 

where  ht  is  the  sampling  period,  k  is  the  iteration  number,  co  is 
the  frequency,  and  the  units  are  centimeters.  Thus,  the  target 
is  translating  in  a  circular  path.  Uniform  image  noise  between 
±0.5  pixel  is  added  to  the  image  features  of  the  target  object. 

4.2.  Controller  Performance 

System  performance  depends  on  the  speed  of  the  target,  the 
type  of  controller  used,  and  the  memory  of  the  controller.  The 
simulated  control  systems  are  distinct  in  two  ways:  (1)  the 
Gauss-Newton  controller  is  non-recursive  or  recursive;  and 
(2)  the  error  velocity  is  uncorrected  or  corrected.  All  schemes 
use  the  estimated  Jacobian  from  the  partitioned  Broyden  up¬ 
date.  The  four  controller  schemes  simulated  here  include: 
(NGN/UV)  non-recursive  Gauss-Newton  controller  given  by 
Algorithm  1  with  uncorrected  error  velocity;  (RGN/UV)  re¬ 
cursive  Gauss-Newton  controller  given  by  Algorithm  2  with 
uncorrected  error  velocity;  (NGN/CV)  non-recursive  Gauss- 
Newton  controller  with  corrected  error  velocity  from  Sec¬ 
tion  3.3;  (RGN/CV)  recursive  Gauss-Newton  controller  with 
corrected  error  velocity  from  Section  3.3. 

The  results  of  a  sample  simulation  utilizing  the  NGN/UV 
controller  {X  =  0.98)  is  shown  in  Figure  1.  The  four  fea¬ 
ture  points  are  seen  initially  in  a  configuration  that  does  not 
correspond  to  the  goal  positions.  The  feature  points  converge 
to  the  goal  positions  and  the  features  remain  at  or  near  the 
goal  positions  even  though  the  target  object  is  moving  in  a 
circular  path  in  the  world  coordinate  frame.  For  a  reference, 
if  a  static  camera  located  at  the  initial  position  the  moving 
camera,  the  speed  of  the  target  object  (moving  at  2.5  cm  s-1) 
roughly  corresponds  to  a  feature  velocity  of  approximately 
63.5  pixel  s-1.  Figure  2  shows  the  initial  positions  of  the  tar¬ 
get  and  the  robot,  and  Figure  3  shows  the  image  error.  The 
average  of  the  four  image  feature  tracking  RMS  errors  is  4.0 
pixels  or  approximately  0.2  cm. 

To  study  the  behavior  of  the  four  different  controllers,  the 
same  simulation  was  repeated  25  times  for  each  controller 
at  eight  different  circular  speeds  co.  Target  velocities  ranged 
from  0.5  to  4  cm  s-1,  or  from  37  to  94  pixel  s-1  for  a  static 
camera.  Each  of  the  25  simulations  has  somewhat  different 
results  due  to  the  addition  of  random  pixel  noise.  Figure  4 
shows  the  average  image  RMS  error  norms  (in  pixels)  for 
the  repeated  trials  using  all  four  controllers.  Outliers  such  as 
those  seen  at  co  =  0.3  and  co  =  0.4  for  the  NGN/UV  indi¬ 
cate  simulations  where  the  system  lost  track  of  the  desired 
pose  and  the  control  became  unstable.  The  NGN/UV  con¬ 
troller  shows  a  distinct  advantage  over  the  other  controllers. 
The  velocity  correction  scheme  does  not  appear  to  offer  any 
advantage.  In  fact,  the  RGN/CV  errors  are  somewhat  worse 
than  the  RGN/UV  errors,  and  the  NGN/CV  simulations  ex¬ 
hibit  complete  loss  of  control.  Although  not  shown,  it  is  in¬ 


teresting  that  the  NGN/CV  simulations  perform  similarly  to 
NGN/UV  simulations  when  the  image  noise  level  is  reduced 
by  half,  indicating  that  it  is  very  sensitive  to  system  noise. 
This  shows  the  beneficial  filtering  action  of  the  recursive  (un¬ 
corrected)  error  velocity  calculation  and  the  amplification  of 
noise  introduced  by  the  first  order  corrector. 

43.  NGN/UV  Performance 

For  the  6-DoF  system  simulated,  the  NGN/UV  controller 
produces  the  best  results.  To  study  the  effects  of  the  for¬ 
getting  factor  X,  a  second  series  of  simulations  was  run  us¬ 
ing  the  NGN/UV  controller  and  varying  the  circular  speed 
co  and  the  forgetting  factor  X.  Figure  5  shows  the  average 
steady-state  image  error  for  co  =  {0.05,  0.1,  0.15,  0.2},  and 
X  =  {0.94,  0.95,  0.96,  0.97,  0.98,  0.99,  1.0}  for  25  simula¬ 
tions  each.  The  average  steady- state  image  RMS  error  for 
each  simulation  is  plotted  by  x  and  the  mean  of  the  averages 
is  plotted  as  a  line.  Clearly  a  long  memory  with  X  >  0.98 
(approximately  20  or  more  significant  samples)  gives  the  best 
overall  performance.  Figure  6  plots  the  mean  of  the  average 
image  error  for  X  =  {0.98,  0.99,  1.00}  of  X  versus  the  target 
speed  determined  by  co.  For  slower  speeds,  a  lower  value  of 
X  produces  results  in  better  tracking  whereas  at  higher  speeds 
X  =  0.9  and  X  =  1  result  in  better  tracking.  These  results 
concur  with  similar  studies  performed  for  the  fixed  camera 
case  in  Piepmeier  (1999). 

4.4.  Additional  Target  Motions 

To  further  validate  the  capabilities  of  the  NGN  controller 
for  uncalibrated  eye-in-hand  visual  servoing,  additional  tar¬ 
get  motions  are  studied.  The  simulations  in  Sections  4.2  and 
4.3  employed  a  simple  circular  path.  This  type  of  path  re¬ 
sults  in  constant  Cartesian  speeds  for  the  target  object.  In  this 
section  two  additional  paths  are  introduced.  First,  we  demon¬ 
strate  tracking  a  square  path  (Figure  7).  Next  we  demonstrate 
tracking  the  target  object  as  it  follows  a  helical  path  (Figure  8) 
involving  target  rotation  and  translation.  The  target  is  twist¬ 
ing  and  retreating  from  the  camera.  To  maintain  the  desired 
pose  between  the  target  and  the  camera,  the  robot  must  move 
the  camera  along  the  robot  base  frame’s  x-axis  as  well  as  fol¬ 
low  the  object’s  rotation  and  circular  motion.  The  square  path 
demonstrates  the  ability  of  the  controller  to  handle  abrupt 
changes  in  target  motion.  The  helical  path  demonstrates  its 
ability  to  follow  targets  through  changing  depths  and  fea¬ 
ture  rotations.  The  target  is  a  5  cm  square  and  the  corners  of 
the  target  object  are  the  four  features  being  tracked.  For  Fig¬ 
ure  7,  the  four  target  features  are  moving  at  3.2  cm  s-1,  and 
for  Figure  8  the  four  target  points  reach  maximum  speeds  of 
(2.9,  1.2,  2.9,  4.0)  cm  s-1.  Figures  9  and  10  show  the  image 
features  for  these  two  paths.  Extension  1  shows  animations 
of  the  robotic  system  as  it  tracks  the  moving  target.  While 
the  paths  are  very  different  in  nature,  the  results  are  nearly 
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Fig.  1.  Image  features  for  eye-in-hand  visual  servoing  using  an  NGN/UV  controller  with  k  =  0.98  and  co  =  0.25  rad  s-1.  A 
moving  object  with  four  features  is  seen  initially  on  the  right.  These  features  converge  on  the  desired  pose  with  an  average 
steady- state  image  RMS  error  of  4.0  pixels. 


Fig.  2.  Plot  showing  the  respective  positions  of  the  target  and  the  robot. 
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Fig.  3.  Image  error  for  using  an  NGN/UV  controller  with  A  =  0.98  and  co  =  0.25  rad  s  1 .  The  average  steady-state  RMS 
error  is  4.0  pixels. 
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Fig.  4.  A  comparison  of  four  different  controllers  at  a  range  of  target  speeds.  In  general,  the  NGN/UV  controller  provides  the 
best  convergent  control. 
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Fig.  5.  A  comparison  of  NGN/UV  controller  performance  showing  image  error  versus  k  for  four  different  target  speeds. 
Individual  errors  are  plotted  for  25  simulations;  the  line  indicates  the  average  error. 


Fig.  6.  The  performance  of  the  NGN/UV  controller  for  k  =  {0.98,  0.99,  1.0}  is  shown  for  eight  different  target  speeds  dictated 
by  cd. 
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Y(m) 

Fig.  7.  Path  taken  by  the  5  cm  square  target  object.  The  object  maintains  a  constant  x  position  at  0.65  m  while  translating  in 
the  y-z  plane.  The  initial  target  position  is  denoted  by  the  lightest  square.  The  darkest  represents  the  final  position. 


X  (m) 


Y  (m) 


Fig.  8.  Path  taken  by  the  5  cm  square  target  object.  The  object  is  located  initially  at  a  depth  of  0.65  m  along  the  x-axis.  The 
object  rotates  as  the  depth  increases  to  0.75  m. 
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Fig.  9.  Image  features  as  seen  by  the  eye-in-hand  camera  following  a  target  object  moving  along  the  path  shown  in  Figure  7. 
Average  steady-state  error  is  4.1  pixels.  The  features  appear  stationary  because  the  robot  maintains  the  desired  pose  relative 
to  the  moving  target. 


Y  (pixels) 


Fig.  10.  Image  features  as  seen  by  the  eye-in-hand  camera  following  a  target  object  moving  along  the  path  shown  in  Figure  8. 
Average  steady-state  error  is  6.0  pixels.  The  features  appear  stationary  because  the  robot  maintains  the  desired  pose  relative 
to  the  moving  target.  Note  the  consistency  between  Figures  1,  9,  and  10  despite  the  differing  target  motions. 


Downloaded  from  http://ijr.sagepub.com  at  US  NAVAL  ACADEMY  LIBRARY  on  December  9,  2008 


8 1 6  THE  INTERNATIONAL  JOURNAL  OF  ROBOTICS  RESEARCH  /  October-November  2003 


identical  to  each  other  and  to  the  results  from  Figures  1-3. 
The  average  steady- state  tracking  error  for  the  square  path 
is  4.1  pixels,  and  the  average  steady-state  tracking  error  for 
the  helical  path  is  6.0  pixels.  The  tracking  errors  are  shown 
in  Figures  11  and  12.  We  see  from  Figures  9  and  11  that  the 
NGN/UV  controller  can  handle  abrupt  changes  in  the  direc¬ 
tion  of  the  target’s  motion.  From  Figures  10  and  12  we  see  that 
the  NGN/UV  controller  can  handle  complex  paths  varying  in 
depth  with  respect  to  the  image  plane. 


5.  Conclusion 

In  this  paper  we  have  investigated  the  use  of  eye-in-hand  vi¬ 
sual  feedback  for  the  tracking  of  moving  targets.  The  methods 
presented  are  uncalibrated  and  require  no  kinematic  models  or 
camera  calibration.  Thus,  the  same  controller  can  be  used  on 
a  wide  variety  of  platforms.  Simulation  results  suggest  that 
a  Gauss-Newton  method  utilizing  a  partitioned  Broyden’s 
method  for  model  estimation  provides  the  best  steady- state 
tracking  behavior.  This  work  is  the  first  presentation  of  an 
uncalibrated,  eye-in-hand,  vision-based  control  scheme  for  a 
moving  target.  While  the  eye-in-hand  system  has  been  studied 
here,  the  algorithm  is  generic,  and  is  applicable  when  the  cam¬ 
era  is  moving  but  not  fixed  to  the  robotic  system.  Furthermore, 
the  control  strategy  could  be  applied  to  either  a  manipulator 
or  mobile  robot  for  uncalibrated  control. 


Appendix  A:  Derivation  of  RLS  Jacobian 
Estimator 

The  recursive  dynamic  Jacobian  estimator  is  the  RLS  solution 
to  the  minimization  problem  (5)  which  is  repeated  here 


fk  —  fo  ~  (4  —  4) 


fk  ~  fk-2  ~  j;  (4  —  h-i) 

fk  ~  fk- 1  -  #  (4  -  4-i)  _ 


diag  hi o 


dia§  hl(k-2) 

fcagKik- 1)  _ 


and  where  I  is  the  identity  matrix,  (Jf)r  denotes  the  rth  col¬ 
umn  of  Jk  and  (diag  hTki)  means  each  element  of  a  diagonal 
matrix  contains  the  row  vector  hTki.  An  overbar  indicates  a 
stacked  and/or  blocked  quantity  so,  for  example,  the  stacked 
vector  e  is  partitioned  into  k  —  1  stacked  vectors  in  ek_x  and 
one  vector  ek ;  the  matrix  W  has  k  blocks  on  the  diagonal. 
Note,  however,  that  xk  merely  stacks  the  columns  of  Jk  into 
a  single  stacked  column. 

The  minimization  problem  is  now  in  the  standard  form  used 
for  a  recursive  solution  (see,  for  example,  Franklin,  Powell, 
and  Workman  1990) 


xk  = 


A  = 


(*4T) i 


(Jl)n 


Ak-i 


*k  —  xk_x  +  Kk  ( bk  —  Akxk_i )  (18) 


min  8k 


A  mki 


k- 1 

HAm«||2 

i= 0 


mk  ( 0i ,  U )  -  mt  ( 0i ,  U) 
dft 

fk~  ft  - —  (4  -  U) 

Ot 


Jkhk 


where  Kk  is  defined  as 

Pk  is  defined  by  the  recursion 


(19) 


where  hki  =  (0*  —  0,-),  0  <  A.  <  1  is  the  weighting  factor,  and 
Jk  is  the  estimated  Jacobian  to  be  determined. 

It  is  direct  to  confirm  that  the  problem  can  be  re-expressed 
in  a  stacked  and  partitioned  form 

min  eT  We 
e  =  b  —  Axk 

where 

Amk0 


Amk{k_2) 

Amk{k_\) 


Pk  =  \(I  -KkAk)Pk_x  (20) 

A 

and  the  initial  values  x0  and  P0  must  be  supplied.  However, 
Pk  also  has  a  non-recursive  definition  that  is  useful  for  deter¬ 
mining  its  symmetric  structure: 


A*-1/  • 

0 

\ 

pk  = 

at 

A 

0 

A.0/ 

/ 

Since  Ak_x  is  a  stacking  of  block  diagonal  matrices,  after 
some  rearranging,  Pk  reduces  to  a  block  diagonal  matrix  with 
repeated  elements 
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Fig.  11.  Image  feature  tracking  error  using  the  NGN/UV  controller  (k  =  0.98)  for  a  target  object  following  the  square  path. 
Average  steady-state  RMS  error  is  4.1  pixels.  The  tracking  is  stable  despite  abrupt  changes  in  the  target’s  motion. 


Fig.  12.  Image  feature  tracking  error  using  the  NGN/UV  controller  (k  =  0.98)  following  a  target  object  moving  along  a 
helical  path.  Average  steady- state  RMS  error  is  6.0  pixels.  The  similarity  to  Figures  3  and  1 1  demonstrates  the  consistency  of 
the  tracking  behavior  despite  the  different  target  motions. 
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Appendix  B:  Index  to  Multimedia  Extensions 


Pk 


Pk  = 


0 


0 

Pk 


where  the  non-recursive  form  of  Pk  is 


A  similar  form  applies  to  Pk_x  which  is  used  to  also  reduce 
Kk  to  a  block  diagonal  matrix  with  repeated  elements 


Kk 


Kk  = 


0 


0 

Kk 


where 


The  multimedia  extension  page  is  found  at  http://www. 
ijrr.org. 

Table  of  Multimedia  Extensions 

Extension  Type  Description 

1  Video  This  file  provides  an  anima¬ 

tion  of  an  eye-in-hand  robotic 
system  as  it  tracks  a  square 
target  moving  along  a  square 
path.  The  left  window  shows 
the  robot  and  target  motion. 
The  right  window  shows  the 
camera  view  as  the  robot  is  ser- 
voing.  The  blue  polygon  rep¬ 
resents  the  desired  view  of  the 
target,  and  the  red  polygon  rep¬ 
resents  where  the  target  is  ac¬ 
tually  seen.  The  target  appears 
as  a  polygon  (and  not  a  square) 
because  the  optical  axis  of  the 
camera  is  not  perpendicular  to 
the  surface  of  the  target  object. 


k(k- 1) 


k(k- 1) 


and  together  with  eqs.  (19)  and  (20)  yields  the  recursive  form 

of  Pk 


Pk  =  \{l-Kkhl(k_X))Pk_ (21) 

Using  these  in  eq.  (18)  and  unstacking  gives 

=  (£_,),  +  Kk  ((bk),  -  i  =  l  ...n 

where  (£*)/  is  the  ith  element  of  bk.  Collecting  the  n  columns 
(Jk)i  and  transposing  gives 

Jk  —  Jk- i  +  (bk  —  Jk-\hk(k-\)^  Kj.  (22) 


Finally,  substituting  for  Kk,  using  the  definition  he  =  hk(k_ i}, 
and  rearranging  the  A  terms  in  eqs.  (22)  and  (21)  yields  the 
desired  results  in  eqs.  (7)  and  (8),  respectively. 
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